Calibration of a 3D camera

ABSTRACT

A method for calibrating a 3D camera includes determining an actuation of an element by a person and calibrating the 3D camera based on the determining of the actuation.

BACKGROUND

Three dimensional Depth-Cameras (3D cameras) which are for example basedon the ToF principle (time-of-flight principle) or other principlesprovide a new technology field with many applications. To give only oneof many applications, 3D cameras may provide human gesture recognitionin natural user interfaces. Such 3D cameras may be for example used formouse replacement in the cubic foot in front of a notebook computer.Distinguished from 2D cameras, 3D cameras provide an array of pixel togenerate for each pixel information related to a distance of the objectcaptured by the pixel. Such information may for example be based on atime of flight of light reflected from an object captured by the pixels,a geometrical relation of light points at the object captured by thepixels or other methods.

3D cameras require a calibration to provide reliable relative andabsolute distance information. Such calibrations are typically made atthe end test of the camera prior to shipping the 3D camera. It would bebeneficial to have a new concept of calibration of 3D cameras which iseasy to implement, provides a high degree of safety or reduces thecalibration time at the end test of the camera.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIGS. 1A and 1B show flow chart diagrams according to embodiments;

FIG. 2 shows a flow chart diagram according to an embodiment;

FIG. 3 shows a schematic drawing of an apparatus according to anembodiment; and

FIG. 4 shows a flow chart diagram according to an embodiment.

DETAILED DESCRIPTION

The following detailed description explains example embodiments. Thedescription is not to be taken in a limiting sense, but is made only forthe purpose of illustrating the general principles of embodiments of theinvention while the scope of protection is only determined by theappended claims.

In the exemplary embodiments shown in the drawings and described below,any direct connection or coupling between functional blocks, devices,components or other physical or functional units shown in the drawingsor described herein can also be implemented by an indirect connection orcoupling unless specifically described otherwise. Functional blocks maybe implemented in hardware, firmware, software, or a combinationthereof.

Further, it is to be understood that the features of the variousexemplary embodiments described herein may be combined with each other,unless specifically noted otherwise.

In the various figures, identical or similar entities, modules, devicesetc. may have assigned the same reference number. Example embodimentswill now be described more fully with reference to the accompanyingdrawings. Embodiments, however, may be embodied in many different formsand should not be construed as being limited to the embodiments setforth herein. Rather, these example embodiments are provided so thatthis disclosure will be thorough and complete, and will fully convey thescope to those skilled in the art. In the drawings, the thicknesses oflayers and regions are exaggerated for clarity.

In the described embodiments, various specific views or schematic viewsof elements, devices, features, etc. are shown and described for abetter understanding of embodiments. It is to be understood that suchviews may not be drawn to scale. Furthermore, such embodiments may notshow all features, elements etc. contained in one or more figures with asame scale, i.e. some features, elements etc. may be shown oversizedsuch that in a same figure some features, elements, etc. are shown withan increased or decreased scale compared to other features, elementsetc.

It will be understood that when an element is referred to as being “on,”“connected to,” “electrically connected to,” or “coupled to” to anothercomponent, it may be directly on, connected to, electrically connectedto, or coupled to the other component or intervening components may bepresent. In contrast, when a component is referred to as being “directlyon,” “directly connected to,” “directly electrically connected to,” or“directly coupled to” another component, there are no interveningcomponents present. As used herein, the term “and/or” includes any andall combinations of one or more of the associated listed items.

It will be understood that although the terms first, second, third,etc., may be used herein to describe various elements, components,regions, layers, and/or sections, these elements, components, regions,layers, and/or sections should not be limited by these terms. Theseterms are only used to distinguish one element, component, region,layer, and/or section from another element, component, region, layer,and/or section. For example, a first element, component, region, layer,and/or section could be termed a second element, component, region,layer, and/or section without departing from the teachings of exampleembodiments.

Spatially relative terms, such as “beneath,” “below,” “lower,” “above,”“upper,” and the like may be used herein for ease of description todescribe the relationship of one component and/or feature to anothercomponent and/or feature, or other component(s) and/or feature(s), asillustrated in the drawings. It will be understood that the spatiallyrelative terms are intended to encompass different orientations of thedevice in use or operation in addition to the orientation depicted inthe figures.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting. As usedherein, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises,”“comprising,” “includes,” and/or “including,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, and/or components.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andshould not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

The embodiments described below are directed to a new concept forcalibrating a 3D camera. In the known art, the 3D camera typicallyundergoes a comprehensive calibration at the end test. All calibrationsteps of 3D cameras are therefore in the known art typically providedonly once prior to the shipping of the 3D camera. A full calibration ofthe 3D camera may include calibrations like absolute distance per pixel,the absolute amplitude per pixel, the dependence of depth measurement onamplitude levels, an identification of dead pixels (out-of-spec-pixels),a shift of above parameters over temperature, a compensation ofintroduced errors of the optical system like errors in illumination,errors introduced in the lense system. The calibration may also addressthe so called wiggling error which may be appear inphoton-mixing-devices based on the time-of flight principle. Thewiggling effect is caused by a non-perfect modulation of the light, i.e.the amplitude modulation of the light transmitted to the object to becaptured has neither a perfect rectangular nor a perfect sinusoidalshape. When a demodulation is done in the pixels of the photon mixingdevices, this effect may cause errors in the phase information andtherefore errors in the depth signal of the 3D camera

As will be described below, with the herein described calibration, atleast a part of the above described calibration processes can be shiftedtowards the final application where the calibration data is generated inreal-time during operation of the 3D-camera. This allows to simplify oreliminate at all the calibration during camera production-test andallows saving test time and production cost at no or very littleexpenses. The calibration described in embodiments allows reducing thecalibration time at the end testing of the device and therefore reducingmanufacturing costs. The concept may provide real time calibrationduring operation of the electronic apparatus in which the 3D camera ismounted. Furthermore, the new calibration concept allows havingrepetitive calibration during operation of the apparatus. Thecalibration can be easily repeated during operation of the apparatus toprovide increased security and confirmation of regular operation of the3D camera. Furthermore, the calibration concept described herein allowsthe calibration at the final application when the 3D camera is mountedin the apparatus.

Referring now to FIG. 1A, a method 100 of calibrating a 3D cameraaccording to an embodiment is described herein. The calibration startsat 102 with determining the actuation of at least one element. Theelement may in some embodiments include a manual actuation element likea key of a keyboard or a manual actuation button. In some embodiments,the element may include other types of elements such as touch screenpanel or a specific region of a touch screen panel configured foractuation of certain functions etc. In some embodiments, the element maybe an actuation element of the apparatus itself in which the 3D camerais mounted. For example, as will be described in FIG. 2, the 3D cameramay be mounted in a portable computing device or Smartphone device andthe element may be an element of the keyboard or a touch pad or asection of a touch screen of the device. In some other embodiments, theelement may be a plug socket, a connector or jack into which specificconnecting elements are to be plugged manually. To give a specificexample, the element may be a headphone plug into which a headphone isplugged. In some embodiments, the element may be an element external tothe apparatus in which the 3D camera is mounted as long as the 3Dcamera's view and position allows taking 3D information of the element.

With reference to 104, the 3D camera is then calibrated based on thedetermining of the actuation. Calibration may include at least one ofthe above described calibration processes for example a calibration ofdepth information such as an absolute distance information provided bythe 3D cameras pixel array or a wiggling error.

With reference to FIG. 1B, an example of the step referenced with 104 inFIG. 1A is described. The calibrating of the 3D camera starts at 12 withobtaining 3D information related to the element. 3D information maytypically be provided in the form of digital frames containing distanceinformation provided by each pixel of the cameras pixel array. Howeverother forms or representation of 3D information may be used in otherembodiments. The 3D information may be raw 3D information or may be 3Dinformation after a digital filter processing. At 14, first distanceinformation related to a distance of the at least one element isextracted from the 3D information. Extracting the distance informationmay include using a gesture control to identify within the 3Dinformation a part of the human body such as an upper part of the user'sfinger which caused the actuation of the element.

At 16, second distance information related to the at least one elementis provided. The second distance information is distance informationwhich is obtained independent of the first information. In other words,the second distance information may be information which is notgenerated by the 3D camera but by any other method or means such as bygeometrical considerations. The second distance information may beprovided for example by using geometric relations which indicate adistance related to the element. For example, a geometricaltriangulation method may be used to determine a distance between the atleast one element and the camera. According to some embodiments, everymethod may be used which is not based on the 3D information provided bythe 3D camera and allows obtaining reliable information on the at leastone element's distance.

The distance between the element and the camera may in some embodimentsbe fixed and the distance information may be preprogrammed or prewrittenin memory of the apparatus. In some embodiments, the distance betweenthe at least one element and the 3D camera may be variable, for exampledue to a mounting of the 3D camera in a moveable or pivotable part ofthe apparatus, or a mounting of the at least one element in a moveableor pivotable part of the apparatus or both. Further, for example whenthe actuation is provided by a touch pad or touch screen, a position atwhich the touch pad element is shown and touched by the user may bevariable dependent for example on a scaling etc. Nevertheless, the touchpad or touch screen can provide exact information for example bycapacitive sensing of the location at which the touch pad was actuatedby the user. Then, by using for example the above described geometricalrelations, the distance between the location at which the user actuatedthe touch pad element and the 3D camera can be calculated.

Referring now to step 18 in FIG. 1B, the calibrating of the 3D camera isperformed based on the first and second information. It is to be notedthat the second information is considered to be the reliable informationand one or more camera parameters may be adapted if the first distanceinformation differs from the second distance information to match the 3Dinformation provided by the 3D camera to obtain the value of the seconddistance information instead of the assumed defective first distanceinformation. In the calibration process, some processing may be providedin order to filter out a systematic or temporarily error orunreliability of the second information. For example, a range ofreliable values for the second information may be defined and thecalibration may be performed only if the second information falls intothis range.

While the method is described in a specific flow chart, it is to beunderstood that some steps may be interchanged without changing theabove described concept.

The calibration described in embodiments uses specific reference points,for example a keyboard, touchpad or other input devices mounted in awell defined or calculatable position in relation to the camera and itsfield of view. At the moment in time when the input device is activated(e.g. one of the keys or buttons etc is pressed) and whenever thisactuation is visible in the field of view of the camera, it may be usedas distance reference for those pixel receiving the fingertip pushingthe key down as the geometric distance between camera and all buttons onthe keyboard is well defined. This process may be continuously performedthroughout the normal operation of the apparatus and with each actuationof the at least one element or with an actuation within a specific timeinterval or based on a predetermined scheme. Such continuousre-calibration may compensate drift effects which may for example becaused by changing temperature or other environmental ornon-environmental effects. The new concept can therefore provide animproved and continuous calibration over the full life cycle of theapparatus compared to the one-time calibration at the end testing of theknown art.

It is to be noted that the 3D camera may in some embodiments be in a lowpower mode and may be instantaneously put into an operating mode basedon the determining that the at least one element has been actuated inorder to allow immediately the providing of the 3D information by the 3Dcamera at a moment when the user's finger is close to the element. Inother embodiments, the 3D camera may be continuously active for exampleto allow tracking of human gestures of the user.

Referring now to FIG. 2, a further embodiment of a calibration method200 will be described. The method starts at 210 with the determiningthat an element has been touched. At 212, 3D information at the momentat which the element is touched is captured by the 3D camera. It is tobe noted that the moment at which the element is touched may also be amoment shortly after or before the exact moment of touching the elementas long as the user's body part which touched the element is within aclose region of the element at this moment in time. At 214, the user'sbody part which touched the element is identified within the 3Dinformation. It is to be noted that also other non-human parts like apart of a connector hold by the user may be detected. Human gesturerecognition processes which may be implemented in the 3D camera anywayto allow tracking of human gestures may be used or partly used toprovide the identification for example of the user's finger or tip ofthe finger within the 3D information or other parts. Thus, the humangesture recognition can be reused during calibration to provide anadditional synergetic effect for calibration.

At 216, based on the identified body part, first distance information isextracted from the 3D information. In some embodiments, the distanceinformation may be distance information of a single pixel of the 3Dcamera pixel array, where the single pixel corresponds to the body parttouching the element. In other embodiments, the distance information maycorrespond to distance information of a group of pixel arrays associatedwith a region of the user's body part or a region around the user's bodypart. In some embodiments, the distance information of the group ofpixel may be processed to obtain the first distance information, forexample by averaging the distance information generated by this group ofpixel. Since the 3D information is captured at the moment when theuser's body part touches the element, the distance of the body part canthus be assumed to be equal or almost equal to the distance of theelement.

At 218, second independent distance information related to a distancebetween the element and the 3D camera is provided. The second distanceinformation may be provided for example by reading a preset orpre-calculated value from a memory or by calculating in real time adistance based on a geometrical configuration of the 3D camera and theelement. Finally, at 220, the calibration of the 3D camera is performedbased on the first and second distance information. It is to be notedthat the second distance information is considered to be reliableinformation for the calibration and therefore adaption of one or more 3Dcamera parameters may be made to match the captured 3D information atthe time of touching to the value of the second distance information.

Referring now to FIG. 3 an example of a portable computing apparatus 300is shown in which the above described methods may be implemented.

FIG. 3 shows the portable computing apparatus 300 including a camera 310mounted in a pivotable cover 312 of the apparatus 300. The cover 312 iscapable to rotate around a pivot 314 provided in a body part 316 of thecomputing apparatus 300. Since a predetermined configuration exists inwhich the 3D camera has a fixed distance to the pivot 314 and a keyboardelement 318 actuated by the user has a fixed distance to the pivot 314,a well defined reference distance is established between a keyboardelement 318 actuated by the user and the 3D camera 312 for each pivotangle between the cover 312 and the body part 316. This distance can becalculated for example by determining the pivot angle and usinggeometrical calculations for the calibration. A wired or wirelessconnection between the keyboard element and an input of the camera maybe established such that the 3D camera can receive signals from thekeyboard element indicating the actuation of the element.

In some embodiments, a calibration of the camera's field of view 320 orother parameters may be performed in addition to the distancecalibration. In such calibrations, information related to a 2D position(x-y-position) or 1D (x-position) of the pixel or pixels which capturethe body part touching the element within the pixel array is provided.With this information calibration of the field of view or other opticalor non-optical parameters may be performed.

FIG. 4 shows a flow chart diagram of a method 400 addressing thecalibration of a field of view or other parameters based on a pixelposition.

At 402 a touching of the element is detected as has already beendescribed above in more detail. At 412, 3D information is captured bythe pixels of the 3D camera at the moment at which the element istouched. At 414, the part which caused the touching of the element isidentified within the 3D information. At 416, information related to alocation of one or more pixels capturing the part within the pixel arrayis provided. At 418 the 3D camera is calibrated based on the informationrelated to a location of the one or more pixels. As described above, theparameters calibrated in 418 may include for example a field of view butalso other optical or non-optical parameters including an opticaldistortion etc.

While some of the above embodiments describe recognition of a user'sbody part, other embodiments may include instead or additional theretoan identification of specific non-human parts which caused a touching ora contact with the element. For example, a headphone connector which isplugged in a headphone jacket may be identified and used for obtainingthe reference distance for the calibration. It is to be understood thatthe gesture recognition processing can be easily adapted to perform suchrecognition processing of non-human parts if the specific form of theseelement is known or can be assumed.

It becomes clear from the above detailed description that the newcalibration of 3D cameras described in embodiments is capable to providea more reliable, easier, and cost-effective calibration of 3D camerascompared to the known art. Furthermore, since in some embodiments noinformation that a calibration is performed is presented to a personwhich caused the actuation of the element, intentional manipulation by auser can be avoided.

In the above description, embodiments have been shown and describedherein enabling those skilled in the art in sufficient detail topractice the teachings disclosed herein. Other embodiments may beutilized and derived there from, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure.

This Detailed Description, therefore, is not to be taken in a limitingsense, and the scope of various embodiments is defined only by theappended claims, along with the full range of equivalents to which suchclaims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

It is further to be noted that specific terms used in the descriptionand claims may be interpreted in a very broad sense. For example, theterms “circuit” or “circuitry” used herein are to be interpreted in asense not only including hardware but also software, firmware or anycombinations thereof. The term “data” may be interpreted to include anyform of representation such as an analog signal representation, adigital signal representation, a modulation onto carrier signals etc.The term “information” may in addition to any form of digitalinformation also include other forms of representing information.

It is further to be noted that embodiments described in combination withspecific entities may in addition to an implementation in these entityalso include one or more implementations in one or more sub-entities orsub-divisions of said described entity. For example, specificembodiments described herein described herein to be implemented in atransmitter, receiver or transceiver may be implemented in sub-entitiessuch as a chip or a circuit provided in such an entity.

The accompanying drawings that form a part hereof show by way ofillustration, and not of limitation, specific embodiments in which thesubject matter may be practiced.

In the foregoing Detailed Description, it can be seen that variousfeatures are grouped together in a single embodiment for the purpose ofstreamlining the disclosure. This method of disclosure is not to beinterpreted as reflecting an intention that the claimed embodimentsrequire more features than are expressly recited in each claim. Rather,as the following claims reflect, inventive subject matter lies in lessthan all features of a single disclosed embodiment. Thus the followingclaims are hereby incorporated into the Detailed Description, where eachclaim may stand on its own as a separate embodiment. While each claimmay stand on its own as a separate embodiment, it is to be notedthat—although a dependent claim may refer in the claims to a specificcombination with one or more other claims—other embodiments may alsoinclude a combination of the dependent claim with the subject matter ofeach other dependent claim. Such combinations are proposed herein unlessit is stated that a specific combination is not intended. Furthermore,it is intended to include also features of a claim to any otherindependent claim even if this claim is not directly made dependent tothe independent claim.

Furthermore, it is intended to include in this detailed description alsoone or more of described features, elements etc. in a reversed orinterchanged manner unless otherwise noted.

It is further to be noted that methods disclosed in the specification orin the claims may be implemented by a device having means for performingeach of the respective steps of these methods.

Further, it is to be understood that the disclosure of multiple steps orfunctions disclosed in the specification or claims may not be construedas to be within the specific order. Therefore, the disclosure ofmultiple steps or functions will not limit these to a particular orderunless such steps or functions are not interchangeable for technicalreasons.

Furthermore, in some embodiments a single step may include or may bebroken into multiple sub steps. Such sub steps may be included and partof the disclosure of this single step unless explicitly excluded.

What is claimed is:
 1. A method for calibrating a 3D camera, the methodcomprising: determining an actuation of at least one element; based onthe determining of the actuation, calibrating the 3D camera.
 2. Themethod according to claim 1, wherein the 3D camera is mounted in anelectronic apparatus, wherein the determining of the actuation of the atleast one element and calibrating the 3D camera are repeated during anormal operation of the electronic apparatus.
 3. The method according toclaim 1, wherein calibrating the 3D camera comprises: identifying anactuation of at least one predetermined element of a plurality ofelements; based on the identifying that the at least one predeterminedelement is actuated, obtaining 3D information, the 3D informationincluding at least 3D information related to the element; identifyingthe at least one predetermined element or a region close to the at leastone predetermined element from the 3D information; extracting from the3D information first distance information related to a distance of theat least one predetermined element; providing second distanceinformation related to a distance of the at least one predeterminedelement, the second distance information being independent from thefirst distance information; and calibrating the 3D camera based on thefirst and second distance information.
 4. The method according to claim3, wherein the information related to a distance of the at least onepredetermined element includes information related to a distance betweenthe at least one predetermined element and the 3D camera.
 5. The methodaccording to claim 3, wherein the identifying of the actuation of the atleast one predetermined element is based on an identifying of theactuation independent from information captured by the 3D camera.
 6. Themethod according to claim 1, wherein no information that a calibrationis performed is presented to a person which caused the actuation of theelement.
 7. The method according to claim 3, wherein the second distanceinformation is determined based on a predetermined configuration of anelectronic apparatus in which the 3D camera is mounted.
 8. The methodaccording to claim 7, wherein the predetermined configuration includes afixed distance of the 3D camera to at least one location in theapparatus and a fixed distance of the predetermined element to the leastone location in the apparatus.
 9. The method according to claim 7,wherein the predetermined configuration includes a predetermineddistance between the 3D camera and the predetermined element.
 10. Themethod according to claim 9, wherein the 3D camera is mounted in firstpart of the apparatus and the element is mounted in a second part of theapparatus, wherein the predetermined distance is based on an anglebetween the first part and the second part of the apparatus.
 11. Themethod according to claim 10, wherein the first part of the apparatus isa cover of the apparatus.
 12. The method according to claim 3, whereinthe identifying of the at least one predetermined element or regionclose to the at least one predetermined element includes identifyingfrom the 3D information a body part which caused the actuation of theelement.
 13. The method according to claim 1, wherein the element is aninput element of the device configured to input information during anormal operation of the device.
 14. The method according to claim 1,further including determining a field of view of the 3D camera based onthe actuation of the element.
 15. A device comprising: a 3D camera; afirst input to receive a first signal indicating an actuation of anelement by a person; wherein the device is configured to calibrate the3D camera in response to the receiving of the first signal indicating anactuation of an element.
 16. The device according to claim 15, whereinthe 3D camera is mounted in an electronic apparatus, wherein the deviceis configured to receive the first signal each time when the element isactuated or according to a predetermined scheme.
 17. The deviceaccording to claim 15, wherein the device is configured to calibrate the3D camera based on identifying an actuation of at least onepredetermined element of a plurality of elements, based on theidentifying that the at least one predetermined element is actuated,obtaining 3D information, the 3D information including at least 3Dinformation related to the element, identifying the predeterminedelement or a region close to the at least one predetermined element fromthe 3D information, extracting from the 3D information first distanceinformation related to a distance of the at least one predeterminedelement based on the identified predetermined element or region close tothe predetermined element, providing second distance information relatedto a distance of the at least one predetermined element, the seconddistance information being obtained independent from the first distanceinformation, and calibrating the 3D camera based on the first and seconddistance information.
 18. The device according to claim 17, wherein theidentifying of the actuation of the at least one predetermined elementis based on an identifying of the actuation independent from informationcaptured by the 3D camera.
 19. The device according to claim 15, whereinthe device is configured to present no information to the person withregards to the calibration being performed.
 20. The device according toclaim 17, wherein the second distance information is determined based ona predetermined configuration of an electronic apparatus in which the 3Dcamera is mounted.
 21. The device according to claim 20, wherein thepredetermined configuration includes a fixed distance of the 3D camerato at least one other location in the apparatus and a fixed distance ofthe predetermined element to the least one other location within theapparatus.
 22. The device according to claim 21, wherein thepredetermined distance includes a predetermined distance between theelement and the 3D camera.
 23. The device according to claim 22, whereinthe 3D camera is mounted in first part of the apparatus and the elementis mounted in a second part of the apparatus, wherein the predetermineddistance is based on an angle between the first part and the second partof the apparatus.
 24. The device according to claim 23, wherein thefirst part of the apparatus is a cover of the apparatus.
 25. The deviceaccording to claim 17, wherein the identifying the at least onepredetermined element or region close to the at least one predeterminedelement from the 3D information includes identifying from the 3Dinformation a part of the body of a person which caused the actuation ofthe at least one predetermined element.
 26. The device according toclaim 15, wherein the device is further configured to determine a fieldof view of the 3D camera based on the actuation of the element.